Situation Monitoring Device and Situation Monitoring System

ABSTRACT

A situation monitoring device which enables monitoring of a variety of situations and reporting in response to the situation using a single device is provided. The situation monitoring device is easy to install and to use and a system therefore can be implemented inexpensively. The situation monitoring device recognizes a place or installation where the device is installed (step S 102 ), holds relational information correlating the place of installation and the situation to be recognized and determines a predetermined situation to be recognized according to place of installation recognition results and the relational information (step S 104 ), recognizes a determined predetermined situation (step S 106 ), and reports the result of the predetermined situation to a user (step S 108 ).

TECHNICAL FIELD

This invention relates to a situation monitoring device that recognizesa situation of a target object and reports that situation, and asituation monitoring system in which such situation monitoring device isconnected to a network, and more particularly, to a situation monitoringdevice and situation monitoring system used for monitoring a situation.

BACKGROUND ART

With advances in continuous internet access and expanded broadbandservice there is a growing awareness of security issues, as evidencedrecently by the commercialization and widespread sale of videocommunications equipment for remote monitoring of homes and offices. Byutilizing these types of existing video communications equipment, it ispossible to construct security systems for observing intrusions bysuspicious persons and monitoring the weak, such as the sick, the aged,and children, from a remote location.

However, with a security system like that described above, it isnecessary for the user at the remote location to check the video dataperiodically, and thus it is difficult to respond quickly when a problemarises. Accordingly, although there is also a security system having theability to detect and report live objects like the system proposed, forexample, by Japanese Laid-Open Patent Publication No. 2002-74566, such asystem provides no more than the ability to detect and report theintrusion by a person who might be a suspicious person.

In addition, with a security system like that described above, due toprivacy concerns arising from the indiscriminate distribution of videodata, the situations to which such a system can be adapted are limited.In order to solve such problems, a specialized system has been proposedthat does not distribute the video itself but instead recognizessituations specified by the user and performs appropriate processingdepending on the situation.

For example, in Japanese Laid-Open Patent Publication No. 2002-352354, asystem that recognizes and reports an emergency situation of a personunder care, based on information such as response by audio or detectionof absence by image recognition, is proposed. In addition, in JapaneseLaid-Open Patent Publication No. 10-151086, a system that recognizes thesituation inside the bathroom of the user from video data and issues awarning when an emergency is detected is proposed.

However, all these systems are constructed as specialized systems forcertain unique situations, and are not a single device capable of beingadapted to a variety of situations. Therefore, for example, whenattempting to construct a security system adapted to a plurality ofobjectives, it is necessary to assemble a plurality of specializeddevices for handling each and every situation, which increases the sizeand the cost of the system. Furthermore, these specialized systems aredifficult to introduce (requiring construction and the like) and are noteasy to install and use. In addition, the composition of a family andthe situations of its members change over time, making these types ofsystems impractical.

By contrast, with recent advances in image processing technology andcalculating power, a great many devices have been proposed thatrecognize ordinary human movements and situations. For example, inJapanese Laid-Open Patent Publication No. 6-251159, a device thatconverts feature vector sequences obtained from time series images intosymbol sequences and selects the most plausible from among the object ofrecognition categories based on a hidden Markov model. In addition, manytechniques for recognizing facial expression have been proposed, such asthe device proposed by Japanese Laid-Open Patent Publication No.11-214316 that recognizes such expressions as pain, excitement and soforth.

However, in attempting to achieve an ordinary movement/situationrecognition device (that is, the capacity to recognize a variety ofsituations using a single device) using these types of techniques, thenumber of mistaken recognitions increases as the categories of movementthat are the object of recognition increase, leading to a furtherincrease in the required processing power.

Furthermore, because these conventional security systems report the samegeneralized emergency target to a predetermined reporting destination(such as a security firm) whenever any sort of emergency arises, it isdifficult to use the device for multiple purposes. For example, in thecase of a security system designed to monitor a child, it is preferablethat the situation of the child be reported to the mother. Similarly, inthe case of a security system designed to monitor emergencies such asthe intrusion of a suspicious person or the outbreak of a fire, it ispreferable that the emergency be reported to the security firm or thelike quickly. However, it has been difficult to get conventionalsecurity systems to operate flexibly according to this sort of widevariety of purposes.

DISCLOSURE OF INVENTION

The present invention is conceived as a solution to the problems of theconventional art, and has as an object to provide inexpensively asituation monitoring device and system configured as a single devicethat that can monitor a variety of situations and report depending onthe situation, and further, that is easy to install and to use.

To achieve the foregoing object, a monitoring device according to thepresent invention has a configuration like that described below, thatis, a situation monitoring device comprising:

place recognition means for recognizing a place of installation wherethe device is installed;

information holding means for holding relational information relatingthe place of installation and a situation to be recognized;

determination means for determining a predetermined situation to berecognized, in accordance with recognition results by the placerecognition means and the relational information;

situation recognition means for recognizing the predetermined situationdetermined by the determination means; and

communications means for reporting the recognition result of thepredetermined situation recognized by the situation recognition means tothe user.

In addition, to achieve the foregoing object, another monitoring deviceaccording to the present invention has a configuration like thatdescribed below, that is, a situation monitoring device comprising:

situation analyzing means for analyzing a situation of a target object;

discrimination means for identifying a predetermined situation fromoutput from the situation analysis means;

situation encoding means configured to convert the situation into apredetermined signal based on the output from the situation analysismeans; and

communications means for reporting the output of the situation analysismeans to the user using the situation encoding means.

According to the present invention, it is possible to provide asituation monitoring device and system configured as a single devicethat that can monitor a variety of situations as well as reportdepending on the situation, and further, that is easy to install and touse.

Other features and advantages of the present invention will be apparentfrom the following description when taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a flow chart illustrating the flow of processing performed bya situation monitoring device according to a first embodiment of thepresent invention;

FIG. 2 is a diagram showing the outlines of the structure of a situationmonitoring system including the situation monitoring device according tothe first embodiment of the present invention;

FIG. 3 is a diagram schematically showing the structure of the situationmonitoring device according to the first embodiment of the presentinvention;

FIG. 4 is a diagram showing the hardware configuration of the situationmonitoring device according to the first embodiment of the presentinvention;

FIG. 5 is a diagram showing a control panel of the controls shown inFIG. 4;

FIG. 6 is a flow chart illustrating details of step S102 shown in FIG.1;

FIG. 7 is a diagram schematically showing image data obtained in stepS602 shown in FIG. 6;

FIG. 8 is a flow chart illustrating details of step S103 shown in FIG.1;

FIG. 9 is a diagram showing sample display contents displayed on an LCDof the controls;

FIG. 10 is a diagram showing a sample recognition information tableindicating the relation between place of installation, a person who isan object of recognition and situation recognition contents;

FIG. 11 is a flow chart illustrating details of step S104 step shown inFIG. 1;

FIG. 12 is a diagram showing sample display contents displayed on theLCD of the controls in step S1103 shown in FIG. 11;

FIG. 13 is a diagram showing the layered structure of the software forthe situation monitoring device;

FIG. 14 is a diagram showing a table indicating the relation betweenlocation code and feature parameters;

FIGS. 15A, 15B and 15C are diagrams schematically showing the structureof a situation monitoring device according to a second embodiment of thepresent invention;

FIG. 16 is a flow chart illustrating the flow of processing performed bythe situation monitoring device according to the second embodiment ofthe present invention;

FIG. 17 is a diagram showing a sample management table;

FIG. 18 is a flow chart illustrating the flow of processing performed bya situation monitoring device according to a third embodiment of thepresent invention;

FIG. 19 is a flow chart illustrating details of step S1802 shown in FIG.18;

FIG. 20 is a diagram showing a sample recognition information tableindicating the relation between a person who is an object of recognitionand situation recognition contents;

FIG. 21 is a diagram showing hardware configuration in a case in which aremote control serves as the controls;

FIG. 22 is a flow chart illustrating the flow of processing of asituation monitoring device according to a third embodiment of thepresent invention;

FIG. 23 is a diagram showing the control panel of the controls shown inFIG. 4;

FIG. 24 is a flow chart illustrating details of a report destinationsetting process (step S2203);

FIG. 25 is a diagram showing a sample report control information table;

FIG. 26 is a diagram showing sample display contents displayed on theLCD of the controls;

FIG. 27 is a diagram showing a sample display of a report destinationsetting screen displayed on the LCD of the controls;

FIG. 28 is a diagram showing a sample conversion table;

FIG. 29 is a diagram showing a table indicating the relation betweenlocation code and feature parameters;

FIG. 30 is a diagram showing the structure of a situation monitoringdevice according to a fourth embodiment of the present invention;

FIG. 31 is a flow chart illustrating details of a report destinationsetting process (step S2203);

FIG. 32 is a diagram showing the contents of the report controlinformation table;

FIG. 33 is a diagram showing an outline of the processing flow of asituation monitoring device according to a fifth embodiment of thepresent invention;

FIG. 34 is a diagram showing a sample report control information table;

FIG. 35 is a diagram showing a sample recognition process softwaremodule provided in step S2205;

FIG. 36 is a flow chart illustrating details of the reporting process(S2209);

FIG. 37 is a flow chart illustrating details of the reporting process(S2209); and

FIG. 38 is a flow chart illustrating details of the reporting process(S2209).

BEST MODE FOR CARRYING OUT THE INVENTION

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

The situation monitoring device according to the present inventionrecognizes predetermined situations of predetermined target objects inresponse to the installation environment of such device and notifies theuser of a change in situation through a network.

First Embodiment

FIG. 2 is a diagram showing the outlines of the structure of a situationmonitoring system, including the situation monitoring device accordingto the first embodiment of the present invention.

In FIG. 2, reference numeral 201 designates a situation monitoringdevice, connected to a network 203 such as the internet by a lineconnection device such as a cable modem/ADSL modem 202. Referencenumeral 204 designates a portable terminal device such as a portabletelephone, which receives situation recognitions results informationthat the situation monitoring device 201 transmits. Reference numeral205 designates a server device having the ability to provide servicessuch as a mail server.

The situation monitoring device 201 generates a text document showingpreviously decided, predetermined information when predetermined changesin situation happen to a target object to be recognized (object ofrecognition) and transmits such information to the mail server 205 as ane-mail document in accordance with an internet protocol. The mail server205, having received the e-mail document, notifies the portable terminaldevice 204 that is the recipient of the e-mail transmission in apredetermined protocol that e-mail has arrived. The portable terminaldevice 204 then accepts the e-mail document held in the mail server 205according to the e-mail arrival information. Thus, a user in possessionof the portable terminal device 204 can confirm a change in situation ofan object of recognition that the situation monitoring device 201detects from a remote location. It should be noted that the situationmonitoring device 201 may be configured so as to have a built-in abilityto access the network 203 directly, in which case the situationmonitoring device 201 is connected to the network 203 without goingthrough the in-house line connection device 202. In addition, theterminal that receives the situation recognition result information isnot limited to the portable terminal device 204, and may be a personalcomputer or a PDA (Personal Digital Assistance), etc.

FIG. 3 is a diagram showing the outlines of the structure of thesituation monitoring device 201 of the first embodiment. In FIG. 3,reference numeral 301 designates a camera lens that tilts (moves up anddown) within a frame designated by reference numeral 302. Referencenumeral 303 designates the outer frame for a pan movement. The lens 301pans (moves left and right) together with such outer frame. Referencenumeral 304 designates a stand, which contains important units otherthan the camera, including the power supply and so forth built in.Consequently, the situation monitoring device 201 can be made compactand lightweight, and moreover, by having a camera that can tilt/panbuilt in, can be easily installed in a variety of different locations.

The user then installs the situation monitoring device 201 in anylocation that suits the purpose and monitors the situation of a giventarget object.

Specifically, the situation monitoring device 201 can be used in avariety of cases, such as the following:

Placed near infants to confirm their safety.

Placed near sick persons to confirm their health.

Placed near the elderly to confirm their safety.

Placed at the entrance of a home to confirm the coming and going offamily members and to monitor the intrusion of suspicious persons.

Placed near windows to monitor the intrusion of suspicious persons.

Placed in the bath to confirm the safety of occupants.

The foregoing is a summary description of the situation monitoringdevice according to the present embodiment and its common uses.Hereinafter, a detailed description is given of the processing performedby such situation monitoring device, with reference to the drawings.

FIG. 4 is a diagram showing the hardware configuration of the situationmonitoring device according to the first embodiment of the presentinvention. In FIG. 4, reference numeral 401 designates a CPU (CentralProcessing Unit), 402 designates a bridge, which has the capability tobridge a high-speed CPU bus 403 and a low-speed system bus 404. Inaddition, the bridge 402 has a built-in memory controller function, andthe capability to control access to a RAM (Random Access Memory) 405connected to the bridge.

A RAM 405 is composed of large-capacity, high-speed memories necessaryfor the operation of the CPU 401, such as SDRAM (Synchronous DRAM)/DDR(Double Data Rate SDRAM)/RDRAM (Rambus DRAM). In addition, the RAM 405is also used as an image data buffer. Furthermore, the bridge 402 has abuilt-in DMAC (Direct Memory Access Controller) function that controlsdata transfer between devices connected to the system bus 404 and theRAM 405. An EEPROM (Electrically Erasable Programmable Read-Only Memory)406 stores a variety of setting data and instruction data necessary forthe operation of the CPU 401. It should be noted that the instructiondata is transferred to the RAM 405 during initialization of the CPU 401,and thereafter the CPU 401 proceeds with processing according to theinstruction data in the RAM 405.

Reference numeral 407 designates a RTC (Real Time Clock) IC, which is aspecialized device for carrying out time management/calendar management.A communications interface 408 is a processor that is necessary toconnect the in-house line connection device (a variety of modems androuters) and the situation monitoring device 201 of the presentembodiment, and may for example be a processor for processing a wirelessLAN (IEEE802.11b/IEEE802.11a/IEEE802.11g and the like) physical layerand lower layer protocol. The situation monitoring device 201 of thepresent embodiment is connected to the external network 203 through thecommunications interface 408 and the line connection device 202.Reference numeral 409 designates controls, and is a processor thatcontrols a user interface between the device and the user. The controls409 are incorporated into a rear surface or the like of the device stand304.

FIG. 5 is a diagram showing a control panel of the controls 409 shown inFIG. 4. Reference numeral 502 designates a LCD that displays messages tothe user. Reference numerals 503-506 designate buttons for menu choices,and are used to manipulate the menus displayed on the LCD 502. Referencenumeral 507, 508 designate an OK button and a Cancel button,respectively. The user sets the situation to be recognized using thecontrol panel 501.

In addition, reference numeral 410 shown in FIG. 4 designates a videoinput unit, and includes photoelectric conversion devices such as CCD(Charge-Coupled Devices)/CMOS (Complimentary Metal Oxide Semiconductor)sensors as well as the driver circuitry to control such devices, thesignal processing circuitry to control a variety of image corrections,and the electrical and mechanical structures for implementing pan/tiltmechanisms. Reference numeral 411 designates a video input interface,which converts raster image data output from the video input unit 410together with sync signals into digital image data and buffers it. Inaddition, the video input interface 411 generates signals forcontrolling the video input unit 410 pan/tilt mechanism.

The digital image data buffered by the video input interface 411 isforwarded to a specific address in the RAM 405 using, for example, theDMAC built into the bridge 402. Such DMA transfer is, for example,activated using the video signal vertical sync signal as a trigger. TheCPU 401 then commences processing the image data held in the RAM 405based on a DMA transfer-completed interrupt signal that the bridge 402generates. It should be noted that the situation monitoring device 201also has a power supply, not shown. This power supply may, for example,be supplied by a rechargeable secondary battery, or, where thecommunications interface 408 is a wire LAN, by Power Over Ethernet(registered trademark).

FIG. 1 is a flow chart illustrating the flow of processing of thesituation monitoring device 201 according to the first embodiment. Thisflow chart is a program loaded into the RAM 405 and processed by the CPU401.

When the situation monitoring device 201 power supply is turned on, instep S101 a variety of initialization processes are carried out.Specifically described, in step S101, an instruction data load (that is,a transfer from the EEPROM 406 to the RAM 405), a variety of hardwareinitialization processes and processes for connecting to the network areexecuted.

Then, in step S102, a process of recognition of the place ofinstallation of such situation monitoring device 201 is executed. In thepresent embodiment, the installation environment in which such device isinstalled is recognized using video image information input by the videoinput unit 410.

FIG. 6 is a flow chart illustrating details of step S102 shown inFIG. 1. First, in a step S601, video data is obtained from the videoinput unit 410 and held in the RAM 405. Next, in step S602, the videoinput interface 411 activates the video input unit 410 pan/tiltmechanism and obtains image data for areas outside the area obtained instep S601. FIG. 7 is a diagram showing schematically image data obtainedin step S602 shown in FIG. 6. The interior of a room is sensed over awide area with the camera image acquisition proceeding in the order ofA->B->C->D.

Then, in step S603, it is determined whether or not the acquisition ofimage data in step S602 is completed. In step S603, if it is determinedthat the acquisition of image data is not completed, processing thenreturns to step S601. By contrast, if in step S603 it is determined thatthe acquisition of image data is completed, processing then proceeds tostep S604.

Then, in step S604, a feature parameter extraction process is performed.It should be noted that it is possible to use a variety of techniquesproposed by the image search algorithm and the like for the process ofextracting a feature parameter. Here, for example, the positiondisplacement feature extraction method of color histograms, higher-orderlocal auto-correlation features (Nobuyuki Otsu, Takio Kurita, SekitaIwao: “Pattern Recognition”, Asakura Shoten, pp. 165-181 (1996)) or thelike is adopted. Specifically, feature parameters that use apredetermined range of color histogram values and local auto-correlationfeatures as features are extracted. Moreover, not only these types ofprimitive features may be used but also higher-level feature extractionmethods may be used as well. For example, a technique may be used inwhich a search is made for particular objects such as a window, bed,chair or desk (K Yanai, K. Deguchi: “Recognition of Indoor Images UsingSupport Relations between Objects”, Transactions of the Institute ofElectronics, Information and Communication Engineers, vol. J84-DII, no.8, pp. 1741/1752 (August 2001)) and the detailed features of thoseobjects (their shape, color, etc.) and the special relations between theobjects are extracted as feature parameters. Specifically, featureparameters that use the presence/position/size/color of the object asfeatures are extracted. It should be noted that, in any case, thefeature parameters are extracted from the image data held in the RAM405.

Then, in step S605, a process of discrimination is carried out using thefeature parameters obtained in step S604 and feature parameterscorresponding to locations already recorded, and a determination is madeas to whether or not the installation environment is a new location inwhich the device has not been installed previously. This determinationis carried out with reference to a table indicating the relation betweenfeature parameters and place of installation. Specifically, where thereexists in the table a place of installation having feature parameters inwhich the Euclidean distance is the closest and moreover exceeding apredetermined threshold, such place of installation is recognized as thelocation where the situation monitoring device 201 is placed. It shouldbe noted that this determination method is not limited to discriminationby distance, and any of a variety of techniques conventionally proposedmay be used.

In step S605, if it is determined that the installation environment is anew location where the device has not been installed previously,processing then proceeds to step S606. By contrast, if in step S605 itis determined that the installation environment is a location where thedevice has been installed previously, processing terminates.

Then, in step S606, location codes corresponding to the featureparameters are registered. FIG. 14 is a diagram showing a tableindicating the correlation between location code and feature parameter.

The “location code” is a number that the device manages. When a newplace is recognized, an arbitrary number not yet used is newlydesignated and used therefore. The “feature parameter” Pnm is scalardata indicating the feature level of a feature m at a location code n.In the case of a color histogram, for example, the Pnm corresponds to anormalized histogram value within a predetermined color range. It shouldbe noted that, for example, this table is held in the EEPROM 406 or thelike.

Thus, as described in the foregoing, in step S102, the device recognizesthe place of installation from the image data and generates both aunique location code that identifies the place of installation andinformation that determines whether or not that location is a newlocation where the device is installed.

Then, in step S103 shown in FIG. 1, the situation to be recognized isdetermined. FIG. 8 is a flow chart illustrating details of step S103shown in FIG. 1.

First, in step S801 in FIG. 8, using the results of the determinationmade in step S102, it is determined whether or not the location wherethe device is installed is a new location where the device has beeninstalled for the first time. If the results of this determinationindicate that the location is new, processing then proceeds to step S802and the operation of setting the object of recognition commences. Bycontrast, if the results of the determination made in step S801 indicatethat the location is not new, processing then proceeds to step S807.

In step S802, the user is prompted, through the controls 409, to set theobject of recognition. FIG. 9 is a diagram showing sample displaycontents displayed on the LCD 502 of the controls 409. If it isdetermined that the location is new, then a message prompting the userto set the object of recognition as described in the foregoing isdisplayed on the LCD 502. When buttons 504-505 are pressed, previouslyregistered persons are displayed in succession. When button 506 ispressed, the person currently displayed is set as the object ofrecognition.

When the selection of the person is completed and the OK button 507 ispressed, the person who is the object of recognition at the currentplace of installation is set in the table (FIG. 10). It should be notedthat, if a person other than one previously registered is selected, thenprocessing proceeds to registration of the person who is the object ofrecognition (905) from a new registration screen (not shown). In theregistration process (905) shown in FIG. 9, video of the person to beregistered is imaged and the feature parameters necessary to recognizesuch registered person is extracted from this video data. Furthermore,in the registration process (905), the user is prompted to enterattribute information for the registered person (such as name, etc.).

FIG. 10 is a diagram showing a sample recognition information tableindicating the relation between the place of installation, the personwho is the object of recognition and the contents of the situation to berecognized. The location code is a unique code assigned to the placerecognized in the place of installation recognition process (step S102).The person code is a unique code assigned to a previously registeredperson. It should be noted that it is also possible to set a pluralityof persons as objects of recognition for a given location (as in thecase of location code P0002 shown in FIG. 10). In this case, an order ofpriority of the objects of recognition may be added to the recognitioninformation table. If an order of priority is set, in the actualrecognition process step the higher the priority of the person the morefrequently he or she is recognized. Furthermore, sometimes a particularperson who is an object of recognition is not set for a given location(as in the case of location code P0003 in FIG. 10).

Next, in step S803, the object of recognition is set. In addition, thedevice determines that there is no change if there is no input for apredetermined period of time, and in step S804 the actual object ofrecognition is determined. Then, in step S804, the recognitioninformation table is checked and the person who is the object ofrecognition is determined. For example, if P002 is recognized as thelocation, then the device recognizes the situations of persons H0001 andH0002. It should be noted that, in the case of a location for which noparticular person is registered as the object of recognition, the devicerecognizes the situations of all persons. For example, the deviceexecutes such recognition processes as detection of entry of allpersons, or detection of all suspicious persons.

By contrast, in step S807, it is determined whether or not the place ofinstallation has been changed. In step S807, if it is determined thatthe place of installation has been changed, processing then proceeds tostep S805. By contrast, if in step S807 it is determined that the placeof installation has not been changed, processing then proceeds to stepS806.

Next, in step S805, through a predetermined user interface, the user isnotified that there has been a change in the place of installation, andfurthermore, the recognition information table is checked and thepersons who are the objects of recognition for the place of installationare similarly reported to the user. Methods that notify and report tothe user through a display on the LCD 502 of the controls 409 or throughvoice information generated by voice synthesis or the like may be usedas the user interface that notifies and reports to the user. Suchprocesses are carried out by the CPU 401.

Next, in step S806, a message concerning whether or not to change thecontents of the setting is displayed for a predetermined period of timeon the LCD 502 of the controls 409, during which time it is determinedwhether or not there has been an instruction from the user to change thetarget object. If the results of the determination carried out in stepS806 indicate that there has been an instruction to change the targetobject, then processing proceeds to step S802 and the object ofrecognition is selected. By contrast, if the results of thedetermination carried out in step S806 indicate there has not been aninstruction to change the target object, processing then proceeds tostep S804. Then, after the object of recognition is determined in stepS804 described above, processing terminates.

Thus, as described in the foregoing, in step S102, the situation to berecognized is determined. Once again, a description is given of theprocess shown in FIG. 1. In step S104 in FIG. 1, the content of thesituation to be recognized is determined. FIG. 11 is a flow chartillustrating details of step S104 shown in FIG. 1.

First, in step S1101, the recognition information table is checked andthe person code of the person who is the object of recognition isacquired from the location code obtained in step S102. In the exampleshown in FIG. 10, when the location code P0002 is recognized, twopersons, with person codes H0001 and H0002, are set as the persons whoare objects of recognition.

Then, in step S1102, it is determined whether or not the content of thesituation recognition at that location has already been set for thesepersons who are objects of recognition. If in step S1102 it isdetermined that the recognition situation at that location has not beenset (as in the case of a new situation), processing then proceeds tostep S1103 and selection of the content of the situation to berecognized is carried out.

FIG. 12 is a diagram showing sample display contents displayed on theLCD 502 of the controls 409 in step S1103 shown in FIG. 11. First, amessage prompting the user to select the content of the situation to berecognized for the designated person is displayed (1201). When buttons504-505 are pressed, preset situation recognition contents are displayedin succession. When button 506 is pressed, the content currentlydisplayed is set as the situation recognition content. When selection ofthe situation recognition content is completed and the OK button 507 ispressed, the situation recognition content for the person who is theobject of recognition at the current place of installation is set in therecognition information table (step S1104). It should be noted that, if“default” (1202) is set or if there is no input from the user after apredetermined period of time has elapsed, then the content isautomatically set to the default. The default is such that a situationordinarily set in most cases, such as recognition of “room entry andexit” and the like, is automatically designated, thereby eliminating theinconvenience attendant upon setting.

By contrast, if the results of the determination carried out in stepS1102 indicate that the content of the situation recognition at thatlocation has already been set, then processing proceeds to step S1108and it is determined whether or not there has been a change in theperson who is the object of recognition. If the results of thisdetermination indicate that there has been in a change in the person whois the object of recognition, processing then proceeds to step S1106. Bycontrast, if the results of the determination carried out in step S1108indicate there has been no change in the person who is the object ofrecognition, processing then proceeds to step S1107.

Then, in step S1106, through a predetermined user interface, the user isnotified that a new person who is the object of recognition has beenset, and furthermore, the recognition information table is checked andthe corresponding situation recognition content is similarly reported tothe user. Methods that notify and report to the user through a displayon the LCD 502 of the controls 409 or through voice informationgenerated by voice synthesis or the like may be used as the userinterface that notifies and reports to the user. Such processes arecarried out by the CPU 401.

Then, in step S1107, a message concerning whether or not to change thecontents of the setting is displayed for a predetermined period of time,during which time it is determined whether or not there has been aninstruction from the user to change the target object. If the results ofthis determination indicate that there has been an instruction to changethe target object, then processing proceeds to step S1103. By contrast,if the results of the determination carried out in step S1107 indicatethat there has not been an instruction to change the target object,processing then proceeds to step S1105.

Then, in step S1103 and step S1104, a process of setting the situationrecognition content is executed as with a new setting. If there is nouser input after a predetermined period of time has elapsed, then thedevice determines that there has been no change in the contents and instep S1105 determines the content of the situation to be actuallyrecognized. Then, in step S1105, the recognition information table ischecked and the situation recognition content for the person who is theobject of recognition is set.

Thus, as described in the foregoing, by the processes of from step S102to step S104 shown in FIG. 1, the person who is the object ofrecognition and the situation recognition content are determined and theactual situation recognition process is executed in accordance with thedetermined conditions.

Next, in step S105, for example, a major change in the background areaof the acquired image data is detected and it is determined whether ornot the place of installation of the situation monitoring device hasbeen moved. This change in the background area can be extracted easilyand at low load using difference information between frames. If theresults of the determination made in step S105 indicate that the placeof installation has changed, then processing returns to step S102 andthe place of installation recognition process is commenced one again. Bycontrast, if the results of the determination made in step S105 indicatethat the place of installation has not changed, processing then proceedsto step S106. Matters are arranged so that this step S105 is executedonly when necessary, and thus the processing load can be reduced.

Next, in step S106 shown in FIG. 1, the person decided upon in step S103is tracked and a predetermined situation of such person is recognized.This tracking process is implemented by controlling the pan/tiltmechanism of the camera through the video input interface 409. In stepS106, for example if P0002 is recognized as the location, the deviceexecutes recognition of the situation, “Have you fallen?” for the personwho is the object of recognition H0001, and executes recognition of thesituation, “Have you put something in your mouth?” for the person who isthe object of recognition H0002. Here, any of the variety of techniquesproposed conventionally can be adapted to that processing relating torecognition of the person which is necessary to this step (e.g., S.Akamatsu: “Research Trends in Face Recognition by Computer”,Transactions of the Institute of Electronics, Information andCommunication Engineers, vol. 80 No. 3, pp. 257-266 (March 1997)). Thefeature parameters needed to identify an individual are extracted duringregistration as described above.

In addition, any of the variety of methods proposed conventionally canbe used for the situation recognition technique processed in step S106.For example, if detecting entry to and exit from a room of a particularperson or detecting the entry into the room of a suspicious person,situation recognition can be easily achieved using the results ofindividual identification performed by a face recognition technique orthe like. Moreover, many methods concerning such limited situations asfeeling ill or having fallen have already been proposed (e.g., JapaneseLaid-Open Patent Publication No. 11-214316 and Japanese Laid-Open PatentPublication No. 2001-307246).

In addition, a situation in which an infant has put a foreign objectinto his or her mouth also can be recognized from recognition of handmovements proposed in conventional sign language recognition and thelike and from information concerning the position of the mouth obtainedby detection of the face. The software that executes the algorithmsrelating to this process of recognition is stored in the EEPROM 406 orthe server device 205 on the network, and is loaded into the RAM 405prior to commencing the recognition process (step S106).

The software for the situation monitoring device 201 according to thepresent embodiment has, for example, a layered structure like that shownin FIG. 13. Reference numeral 1301 designates an RTOS (Real TimeOperating System), which processes task management, scheduling and soforth. Reference numeral 1302 designates a device driver, which, forexample, processes device control of the video input interface 411 orthe like. Reference numeral 1303 designates middle ware, and processessignals and communications protocols relating to the processes performedby the present embodiment. Reference numeral 1304 designates applicationsoftware. The software necessary for the situation recognition processesrelating to the present embodiment is installed as the middle ware 1303.The software with the desired algorithm is dynamically loaded andunloaded as necessary by a loader program of the CPU 401.

Specifically, when the situation to be recognized is determined in stepS1105, in the example described above two types of processing softwaremodels recognizing the situation “Has person fallen?” for person H0001and the situation “Has person put something in your mouth?” for personH0002 are loaded from the EEPROM 406. By limiting the recognitionsituation by the device installation environment or the person who isthe object of recognition, complication of the recognition processalgorithm can be avoided and a practical system can be builtinexpensively.

In addition, it is also possible to provide inexpensively a system witheven greater expandability by storing this type of processing softwareon another server device connected to the network. In this case, whenthe content of the situation to be recognized is determined (stepS1105), the CPU 401 accesses the prescribed server device and forwardsthe prescribed software modules from the server device to the RAM 406using a communications protocol such as FTP (File Transfer Protocol) orHTTP (Hyper Text Transfer Protocol). In step S106 shown in FIG. 1, suchsoftware is used as situation recognition process software. By storingthe processing software modules on the server device, the capacity ofthe EEPROM 406 can be reduced, and moreover, device function expansion(processing algorithm expansion) can be easily achieved.

Then, in step S107 shown in FIG. 1, a determination is made as towhether or not the predetermined situation had been recognized. If theresults of this determination indicate that such a predeterminedsituation has been recognized, processing then proceeds to step S108 andthe CPU 401 executes a reporting process. This reporting process may,for example, be transmitted as character information through thecommunications interface 408 according to e-mail, instant messaging orsome other protocol. At this time, in addition to character information,visual information may be forwarded as well. In addition, the device maybe configured so that, if the user is in the same house where the deviceis installed, the user may be notified of the occurrence of an emergencythrough an audio interface, not shown.

By contrast, if the results of the determination made in step S107indicate that the predetermined situation has not been recognized, thenprocessing returns to step S105 and a check is made to determine thepossibility that the place of installation has been moved. If the placeof installation has not changed, the situation recognition process (stepS106) continues.

Thus, as described above, in the present embodiment, in accordance withthe results of the recognition of the place of installation thesituation monitoring device, the situation to be recognized and theperson who is to be the object of recognition are determinedautomatically, and furthermore, the appropriate recognition situation isset automatically in accordance with the results of the recognition ofthe person who is the object of recognition. Consequently, it becomespossible to implement an inexpensive situation monitoring device thatuses few resources. In addition, merely by placing the device in anarbitrary location, a situation monitoring capability can be providedthat is suitable for that location, and since a single device handles avariety of situations it is convenient and simple to use.

Second Embodiment

FIGS. 15A, 15B and 15C are diagrams schematically showing the structureof a situation monitoring device according to a second embodiment of thepresent invention. Reference numeral 1501 shown in FIG. 15A designatesthe main part of the situation monitoring device, containing thestructure shown in the first embodiment. Reference numerals 1502 a-1502c shown in FIGS. 15A-15C designate a stand called a cradle, with themain part set in the cradle. To the main part 1501 is attached aninterface for supplying power from the cradle 1502 and an interface forinputting information. The cradle 1502 is equipped with a device thatholds information for uniquely identifying the power supply and thecradle. An inexpensive information recording device such as a serial ROMcan be used as that device, and can communicate with the main part 1501through a serial interface.

The processing operation performed by the situation monitoring device ofthe second embodiment differs from the processing operation performed bythe first embodiment only in the process of step S102 shown in FIG. 1.

FIG. 16 is a flow chart illustrating the flow of processing performed bythe situation monitoring device according to the second embodiment.

First, in a step S1601, the CPU 401 accesses the serial ROM built intothe cradle 1502 through a serial interface, not shown, and reads out IDdata recorded on the ROM. Here, the read-out ID code is a unique codethat specifies the place of installation. Then, in step S1602, a tablethat manages the ID code is checked.

Then, in step S1603, it is determined whether or not the place ofinstallation of that ID code is a new location. It should be noted thatthe management table is assumed to be stored in the EEPROM 406. FIG. 17is a diagram showing a sample management table, in which ID codescorresponding to arbitrary location codes that the situation monitoringdevice manages are recorded. If the results of the determination made instep S1603 indicate that the place of installation of the ID code is anew location, then processing proceeds to step S1604 and that ID code isrecorded in the management table in the EEPROM 406. By contrast, if theresults of the determination made in step S1603 indicate that the placeof installation of the ID code is not a new location, processing thenproceeds to step S1604.

In the case of the present embodiment, by setting the main part 1501 onthe cradle 1502, the cradle so set is recognized, and consequently, thelocation where the device is installed is recognized. It should be notedthat the processing steps that follow the place of installationrecognition process (step S102) are the same as those of the firstembodiment, with the object of recognition and the situation to berecognized determined according to the location.

In addition, in the case of the present embodiment, the user installs inadvance cradles in a plurality of locations where the situationmonitoring device is to be used and moves only the main part 1501according to the purpose for which the device is to be used. Forexample, cradle 1502 a is placed in the entrance hallway and cradle 1502b is placed in the children's room. Accordingly, if, for example, themain part 1501 is set on the cradle 1502 a, the device operates in asituation recognition mode that monitors for entry by suspiciouspersons, and if set on the cradle 1502 b, the device operates in asituation recognition mode that monitors the safety of the children.

As is clear from the foregoing description, according to the secondembodiment, the place of installation can be recognized accurately byusing a simple method in which the location is recognized by acquiringan ID code.

Third Embodiment

FIG. 18 is a flow chart illustrating the flow of processing performed bya situation monitoring device according to a third embodiment of thepresent invention. The flow chart is a program loaded into the RAM 405,and processed by the CPU 401. In the case of the present embodiment aswell, the hardware configuration is the same as that of the firstembodiment of the present invention, and thus a description is given ofonly that which is different from the first embodiment.

When the power to the situation monitoring device is turned on, in stepS1801 a variety of initialization processes are executed. Specifically,in step S1801, processes are executed for loading instruction data(forwarding data from the EEPROM 406 to the RAM 405), initialization ofhardware, and network connection.

Then, in step S1802, the content of the object of recognition and thesituation to be recognized for that object of recognition are selected.FIG. 19 is a flow chart illustrating details of step S1802.

In step S1901, the user is prompted to set the object of recognitionthrough the controls 409. FIG. 9 is a diagram showing sample displaycontents displayed on the LCD 502 of the controls 409. First, a messageprompting the user to select an object of recognition is displayed(901). When buttons 504-505 are pressed, previously registered personsare displayed in succession. When button 506 is pressed, the personcurrently displayed is set as the object of recognition.

When the selection of the person is completed and the OK button 507 ispressed, the person who is to be the object of recognition at thecurrent place of installation is recorded in the table (step S1902). Itshould be noted that, if a person other than one previously registeredis selected, then, as with the first embodiment, the device enters amode of registering the person who is to be the object of recognitionfrom the new registration screen 905.

FIG. 20 is a diagram showing a sample recognition information tableshowing the relation between a person who is the object of recognitionand a situation to be recognized.

The codes for the person who is the object of recognition are uniquecodes assigned to previously registered persons. In addition, codeshaving a special meaning can be assigned to the person who is the objectof recognition. For example, in the example shown in FIG. 20, H9999 is aspecial code indicating that all persons are targeted. When such a codeis selected, a predetermined situation is recognized for all persons.

Then, in a step S1903, the type of person selected as the object ofrecognition as well as the situation recognition content are reported tothe user. Methods that notify and report to the user through a displayon the LCD 502 of the controls 409 or through voice informationgenerated by voice synthesis or the like may be used as the userinterface that notifies and reports to the user.

In step S1905, a display querying the user whether or not the selectedcontent of the situation recognition is to be changed is carried out fora predetermined period of time, and a determination is made as towhether or not there has been an instruction from the user to change theselected content of the situation recognition within the predeterminedperiod of time. If the results of this determination indicate that therehas been an instruction from the user to change the selected content ofthe situation recognition, processing then proceeds to step S1906. Bycontrast, if the results of that determination indicate that there hasbeen no instruction from the user to change the selected content of thesituation recognition, then processing terminates.

Then, in step S1906, the content of the situation to be recognized foreach person who is the object of recognition is set. For example, whenthe buttons 504-505 are pressed, preset situation recognition contentsare displayed in succession. When button 506 is pressed, the contentcurrently displayed is set as the situation recognition content. Whenselection of the situation recognition content is completed and the OKbutton 507 is pressed, the situation recognition content for the personwho is the object of recognition at the current place of installation isset in the recognition information table (step S1104). It should benoted that, if “default” (1202) is set or if there is no input from theuser after a predetermined period of time has elapsed, then the contentis automatically set to the default. The default is such that asituation ordinarily set in most cases, such as recognition of “roomentry and exit” and the like, is automatically designated, therebyeliminating the inconvenience attendant upon setting.

When setting of the situation recognition content is completed, theactual recognition operation is commenced. First, in step S1803 shown inFIG. 18, the process of detecting and recognizing the object ofrecognition is carried out. Here, too, as described with respect to thefirst embodiment, any conventionally proposed person recognitionalgorithm or the like can be used for the process of recognizing thetarget object. It should be noted that if the person detected is a newperson not set in the recognition information table, then the process ofsetting the person in the recognition information table is carried outin the setting step (S1802). However, in step S1804 the determinationwhether or not to move to the setting process can be set in advance bythe user. That is, when a person not set in the table is detected, it isalso possible to set the device to routinely ignore that person or carryout previously determined default situation recognition.

Then, in step S1805, the recognition information table is checked andthe situation recognition content for the recognized person isdetermined. Then, in step S1806, the situation recognition process forthe situation recognition content determined in step S1805 is executed.As with the first embodiment, the situation recognition performed herecan also be accomplished using any of the variety of methods proposedconventionally. Then, in step S1807, when it is determined that apredetermined recognition of a predetermined person has been identified,as with the first embodiment, in step S1808, the user is notified.

Thus, as described above, with the third embodiment, the situation to berecognized is automatically determined for each person who is the objectof recognition and an appropriate situation recognition is automaticallyset. Consequently, it is possible to implement an inexpensive systemthat uses few device resources. In addition, merely by placing thedevice in an arbitrary location, a situation monitoring capability canbe provided that is suitable for that location, and since a singledevice handles a variety of situations it is convenient and simple touse.

It should be noted that, although the foregoing embodiments aredescribed in terms of a person who is the object of recognition, thepresent invention is not limited to such a situation and may, forexample, be adapted to any object of recognition, such as an animal or aparticular object, etc. For example, in the case of a particular object,the device may be used to recognize and report such situations as thatsuch object “has been moved from a predetermined position” or “has gonemissing”. Recognition of movement or presence can be accomplished easilyby using a pattern matching technique proposed conventionally.

In addition, although the foregoing embodiments are described in termsof recognizing the location where the device is installed and thesituation of the object of recognition target using video information,the present invention is not limited thereto and may, for example, beconfigured so as to recognize situation using sensing information otherthan video information. Furthermore, the present invention may use acombination of video information and other sensing information.Information gathered by voice, infrared, electromagnetic wave or othersuch sensing technologies can be used as the sensing information.

In addition, although the foregoing embodiments are described in termsof defining the relation between the place of installation, the objectof recognition and the situation recognition content using an ordinarytable, the present invention is not limited thereto and may, forexample, make determinations using higher level recognitiontechnologies. For example, a technique may be used in which high-leveldiscrimination is carried out concerning the significance of a location(i.e., that the place is a child's room or a room in which a sick personis sleeping) from the recognition of particular objects present at theplace of installation or the identification of persons appearing at suchlocation, and using the results of such recognition and identificationto determine the object of recognition and the situation recognitioncontent.

In addition, although the first embodiment described above is describedin terms of commencing the process of recognition of the place ofinstallation of the device using a change in the acquired background,the present invention is not limited thereto and may, for example, useother techniques. For example, a method may be used in which amechanical or an optical sensor is attached to the bottom of the devicethat detects when such device is picked up and later set down again,with location recognition commenced at such times. Moreover, a methodmay be used in which the process of recognizing the location iscommenced when a predetermined button on the controls is set. In eithercase, the processing load can be reduced compared to executing thelocation recognition process continuously. Furthermore, a method likethat in which the location recognition process is commencedautomatically at predetermined time intervals using the RTC 407 may beused. In this case as well, the processing load can be reduced comparedto executing the location recognition process continuously.

In addition, although the second embodiment described above is describedin terms of recognizing the place of installation by the differentcradles on which the situation monitoring device is set, the presentinvention is not limited thereto and may, for example, use othertechniques. For example, the device may be given a built-in wireless tagreceiver so that, for example, the place of installation of the devicemay be detected by detecting a wireless tag affixed at a predeterminedlocation within the house. In this case, the wireless tag can beprovided by a seal or the like, thus making it possible to implement,easily and inexpensively, a reliable place of installation detectioncapability. Furthermore, the device may be given a built-in, independentposition information acquisition unit in the form of a GPS (GlobalPosition System) or the like, and the information obtained by such unitused to acquire the position of the device inside the house, etc. Inthis case, by combining GPS position detection results and imagedetection results, it is possible to provide a more accurate place ofinstallation recognition capability.

In addition, although the foregoing embodiments are described in termsof using internet e-mail as a medium of reporting a change in thesituation of the object of recognition, it is conceivable that problemsmight occur with real-time transmission if e-mail protocols are used.Accordingly, other protocols may be used. For example, by using instantmessaging protocol and the like, it is possible to achieve rapidinformation reporting. Moreover, the invention may be configured sothat, instead of reporting by text message, the device main unit isprovided with a built-in telephone capability and voice synthesiscapability, so as to contact the remote location directly by telephoneto report the information.

In addition, although the foregoing embodiments are described in termsof using a camera having a mechanical control structure (a so-calledpan/tilt camera), the present invention is not limited thereto and may,for example, employ a wide-angle camera instead. In that case, theobject of recognition is not supplemented mechanically but instead anequivalent process can be implemented using image data acquired at wideangles.

In addition, although the foregoing embodiments are described in termsof providing the device main unit with a control unit having aninput/output capability as the controls, the present invention is notlimited thereto and may, for example, employ a remote control or thelike that is separate from the device as the control unit. FIG. 21 is adiagram showing the hardware configuration in a case in which a remotecontrol is used for the control unit. In FIG. 21, only the controls 2109are different from the hardware configuration described with respect tothe first embodiment above (FIG. 4). Thus, reference numerals 2109 b, cdesignate communications units for controlling communications betweenthe controls I/F 2109 and the main unit, implemented using a wirelessinterface such as an electromagnetic wave or infrared wirelessinterface. These communications units can be implemented easily andinexpensively using low-speed wireless transmission medium. Referencenumeral 2109 a designates the controls I/F, which is equipped withdisplay/input functions like the controls 409 shown in the firstembodiment. A remote control 2109 d, consisting of the controls I/F 2109a and the communications unit 2109 b, is lightweight and compact. Theuser can set parameters needed for the operation of the device byoperating the remote control 2109 d. Separating the controls from themain unit in the foregoing manner provides greater flexibility in theinstallation of the device and enhances its convenience as well.

Furthermore, the invention may be configured to set the parametersneeded for operation using a network. For example, the invention may beprovided with an HTTP (Hyper Text Transfer Protocol) server capabilityand the user provided with a Web-based user interface based on HTTP viaa communications interface 2108. The HTTP server may be incorporated asone part of the middle ware (reference numeral 1303 shown in FIG. 13),activating a predetermined parameter setting program in response toinput from the remote location based on HTTP. The user is able to setthe parameters needed for operation of the main unit from an ordinaryterminal such as a mobile telephone, a PDA, a personal computer or thelike. Furthermore, such setting operation can be carried out from theremote location. Moreover, the device can be implemented inexpensivelybecause it does not require provision of a special control unit.

In addition, although the foregoing embodiments are described in termsof executing all processes using a processor incorporated in and builtinto the situation monitoring device, the present invention is notlimited thereto and may, for example, be implemented in combination witha personal computer or other such external processing device. In thatcase, only the reading in of image data is accomplished using a specialdevice, with all other processing, such as image recognition,communications and so forth, accomplished using personal computerresources. By using a wireless interface such as BlueTooth, for example,or a power line communications interface such as HPA (Home Power PlugAlliance) or the like to connect the specialized device and the personalcomputer, the same convenience as described above can be achieved. Thissort of functionally dispersed situation monitoring system can of coursebe achieved not only with the use of a personal computer but also withthe aid of a variety of other internet appliances as well.

In addition, although the foregoing embodiments are described in termsof implementing the present invention by software processing using aCPU, the present invention is not limited thereto and may, for example,be implemented by special hardware processing as well. In that case, thealgorithm for situation recognition corresponds to object data thatdetermines the internal circuitry of an FPGA (Filed Programmable GateArray) or object data that determines the internal circuitry of areconfigurable processor. When the situation to be recognized isdetermined (step S1105), the system control processor loads the datafrom the EEPROM 406 or a server device connected to the network or thelike into the special hardware. The special hardware then commencesrecognition processing of a predetermined algorithm according to theobject data that has been loaded.

Thus, as described above, according to the present embodiments, becausethe content of the situation to be recognized is limited depending onthe place of installation of the device itself, it is possible toachieve a reliable situation monitoring device inexpensively. Moreover,because the place of installation is diagnosed automatically and theappropriate situation to be recognized is determined accordingly, theuser can recognize a variety of situations simply by installing a singledevice.

In addition, according to the above-described embodiments, because theobject of recognition and the situation recognition content are limitedaccording to the place of installation of the device, it is possible toachieve a more reliable situation monitoring device inexpensively.Moreover, because the place of installation is diagnosed automaticallyand the appropriate object of recognition and situation to be recognizedare determined accordingly, the user can recognize a desired situationwith a high degree of reliability simply by installing the device.

In addition, according to the above-described embodiments, because thesituation recognition content is limited according to the object ofrecognition, it is possible to achieve a reliable situation monitoringdevice inexpensively. Moreover, the user can recognize a desiredsituation simply by placing the device near the target object ofrecognition or a location where there is a strong possibility that thetarget object of recognition will appear.

In addition, according to the above-described embodiments, the devicecan be implemented inexpensively without the need for special sensorsand the like. Moreover, carrying out location recognition processingonly where necessary enables the processing load to be reduced. As aresult, location recognition processing can be commenced reliably withan even simpler method. Furthermore, location recognition processing canbe commenced reliably without the addition of special sensors and thelike.

Moreover, it is possible to prevent errors in the recognition functionproduced by erroneous recognition of the place of installation. It isalso possible to prevent errors in the recognition function produced byerroneous recognition of the object of recognition. It is also possibleto provide a user interface for setting information at the appropriatetime, thus improving convenience.

In addition, according to the above-described embodiments, because it ispossible to provide a user interface for setting informationautomatically when changing the place of installation, thus improvingconvenience. It is also possible to provide a user interface for settinginformation only when changing the place of installation, and even thenonly when necessary, thus improving convenience. It is also possible toprovide a user interface for setting information only when necessary,depending on the results of the recognition of the object ofrecognition.

In addition, according to the above-described embodiments, providing auser interface for setting information only when necessary improvesconvenience and makes it possible to achieve more desirable situationrecognition depending on the order of priority. It is also possible torecognize the place of installation of the device reliably using asimple method.

In addition, the above-described embodiments make it more convenient forthe user to set the parameters necessary for operation of the device,and also enable the user to set the parameters necessary for theoperation of the device from a remote location. It is also possible toset the parameters necessary for the operation of the device from anordinary terminal. In addition, it is possible to achieve a morecompatible device with greater expansion capability inexpensively.

Fourth Embodiment

FIG. 22 is a diagram showing the outlines of a processing flow performedby a situation monitoring device according to a fourth embodiment of thepresent invention. Such processing flow is a program loaded in the RAM405 and processed by the CPU 401.

When the situation monitoring device 201 power supply is turned on, instep S2201 a variety of initialization processes are carried out.Specifically, instruction data load (that is, a transfer from the EEPROM406 to the RAM 405), hardware initialization and connection to thenetwork are executed.

Next, in a step S2202, a process of identifying the place ofinstallation is executed. In the present embodiment, the place ofinstallation of the device is identified using video image informationinput using the video input unit 410. It should be noted that thedetails of the place of installation identification process (step S2202)are the same as those described in FIG. 6 with respect to the firstembodiment described above, and thus a description thereof is omittedhere (the table indicating the relation between the location codes andthe feature parameters are the same as in FIG. 14 (see FIG. 29)).

Alternatively, instead of performing the identification of the place ofinstallation automatically, the device may be configured so that theuser performs this task manually. In that case, the user inputsinformation designating the place of installation through an interface,not shown, displayed on the control panel 501 of the controls 409.

In addition, when selecting the destination for the reporting of thesituation recognition content or the reporting medium, when not usinginformation relating to the place of installation, the place ofinstallation identification process (step S2202) or the place settingprocess may be eliminated.

Next, in step S2203, the destination of the reporting when apredetermined situation is recognized is set. FIG. 24 is a flow chartillustrating details of a report destination setting process (stepS2203).

In step S2401, an interface, not shown, querying the user whether or notto change the settings is displayed on the control panel 501 of thecontrols 409. In the event that the user does change the settings, thesetting information stipulating the reporting destination is updated inthe steps (S2402-S2405) described below.

First, in step S2402, the user is prompted to set the object ofrecognition through the controls 409 (reference numeral 901 in FIG. 9).It should be noted that FIG. 9 shows sample display contents displayedon the LCD 2301 (FIG. 23) of the controls 409.

Here, when buttons 504-505 are pressed, previously registered personsare displayed in succession (902-904). When button 506 is pressed, theperson currently displayed is set as the target of a reporting eventoccurrence. When selection of the situation recognition content iscompleted and the OK button 507 is pressed, the person who is the objectof recognition at the current place of installation is set in areporting control information table (FIG. 25).

The reporting control information table is table data stored in theEEPROM 406 or the like, and is checked when determining a reportingdestination to be described later. In other words, the reportingdestination during a reporting event occurrence is controlled bychecking this table. It should be noted that, when a person other thanone previously registered is selected, then processing proceeds toregistration of the person who is the object of recognition (905) from anew registration screen (not shown). In the registration process (905),video of the person to be registered is imaged and the featureparameters necessary to recognize such registered person is extractedfrom this video data. Furthermore, in the registration process (905),the user is prompted to enter attribute information for the registeredperson (such as name, etc.).

FIG. 25 shows a sample reporting control information table showing therelation between a person who is the object of recognition, the contentof the reporting and the reporting destination. The location code is aunique code assigned to the location recognized in the place ofinstallation recognition step S2202. The person code is a unique codeassigned to previously registered persons.

It should be noted that it is also possible to establish a plurality ofpersons as the object of recognition for a location (as in the case oflocation code P0002 shown in FIG. 25). In this case, an order ofpriority of the objects of recognition may be added to the reportingcontrol information table. If an order of priority is established, thenin a process of analyzing the content of the situation (step S2205) thesituation of a person of higher priority is subjected to recognitionprocessing more frequently. Furthermore, sometimes a particular personwho is an object of recognition is not set for a given location (as inthe case of location code P0004 in FIG. 25). In this case, when apredetermined situation at that location is recognized (such asintrusion by a person), the reporting process is executed in step S2209regardless of the output of the object recognition process of stepS2206.

Next, in step S2403, the content of the situation for which reporting isto be carried out is set for each person who is the object ofrecognition. FIG. 26 shows one example of display contents displayed onthe LCD 2301 of the controls 409. When buttons 504-505 are pressed,previously registered recognition situation contents are displayed insuccession. When button 506 is pressed, the situation currentlydisplayed is set as the reporting occurrence situation for that personwho is the object of recognition object of recognition.

When selection of the situation content is completed and the OK button507 is pressed, the situation content at the current place ofinstallation is set in the reporting control information table (FIG.25). It should be noted that when the “default” (2602) is set or whenthere is no input from the user for a predetermined period of time, thecontent is automatically set to the default setting. The default is suchthat a situation ordinarily set in most cases, such as recognition of“room entry and exit” and the like, is automatically designated, therebyeliminating the inconvenience attendant upon setting.

Next, in step S2404, the reporting destination for the reporting is setfor each object of recognition and its situation content. FIG. 27 showsa sample display of a reporting destination setting screen displayed onthe LCD 2301 of the controls 409. When buttons 504-505 are pressed,previously registered reporting destinations are displayed insuccession. When button 506 is pressed, the reporting destinationcurrently displayed is set as the reporting destination when a situationof the person who is object of recognition is recognized.

When selection of the situation to be recognized is completed and the OKbutton 507 is pressed, the reporting destination is set in the reportingcontrol information table (FIG. 25). It should be noted that, if a “newregistration” (2705) is set, then a predetermined interface, not shown,is displayed on the predetermined control panel 501 and registration ofa new reporting destination is carried out. In addition, it is alsopossible to set a plurality of reporting destinations for a singlesituation.

As described above, in steps S2402-S2404 the reporting controlinformation table (FIG. 25) for a given location is set. To explain inspecific terms using FIG. 25, if the location code is P0002, the query“Has person fallen?” is set as the reporting condition for person H1001and a report to that effect is made to “Father” if that condition isrecognized.

In addition, the queries “Has person put something in his mouth” and “Isperson in a prohibited area?” are set as reporting conditions for personH1002, and reports are made to that effect to “Mother” and “OlderBrother” if situations of such conditions are recognized. It should benoted that in the case of locations for which no particular persons areregistered, the system recognizes the situations of all persons or thesituation of that location (such as the outbreak of a fire and soforth). For example, in FIG. 25, at location P0004, such recognitionprocesses as detection of the entry of all persons or detection of asuspicious person are executed and a report to that effect is made to“Security Company” if intrusion by a person is detected.

As described above, in step S2203, the object of recognition, thesituation to be recognized and the corresponding reporting destinationare recorded in the reporting control information table.

Next, in step S2204, it is determined whether or not there has been achange in situation. Here, for example, using the difference betweenframes of image data, the system detects changes in image in the area ofthe object of recognition. If a change beyond a predetermined area isconfirmed in this step, then in step S2205 the process of analyzing thecontent of the situation of the target object is commenced. It should benoted that, in step S2204, for example, a change in situation may bedetected using information other than image data. For example, atechnique may be used in which intrusion by a person is detected using asensor that uses infrared rays or the like. In this step, a change inthe situation (such as the presence of a person) is detected with asimple process and the process of analyzing the content of the situation(step S2205) is executed only when necessary.

When a change in situation is detected, in step S2205 the process ofanalyzing the change in situation is executed. In step S2205, a personwithin the sensing range is tracked and the situation of that person isanalyzed. It should be noted that it is possible to utilize any of thevariety of methods proposed conventionally for the necessary situationrecognition technique. For example, detection of the entry into a roomof a particular person or the entry of a suspicious person into the roomcan be accomplished easily using individual identification resultsproduced by face detection/face recognition techniques. In addition,many techniques for recognizing facial expression have been proposed,such as the device proposed by Japanese Laid-Open Patent Publication No.11-214316 that recognizes such expressions as pain, excitement and soforth.

Furthermore, a situation in which an infant has put a foreign objectinto his or her mouth also can be recognized from recognition of handmovements proposed in conventional sign language recognition and thelike and from information concerning the position of the mouth obtainedby detection of the face. Furthermore, in Japanese Laid-Open PatentPublication No. 6-251159, a device that converts feature vectorsequences obtained from time series images into symbol sequences andselects the most plausible from among the object of recognitioncategories based on a hidden Markov model is proposed.

In addition, in Japanese Laid-Open Patent Publication No. 01-268570, amethod of recognizing a fire from image data is proposed. In step S2205,processing modules including this plurality of situation recognitionalgorithms are executed, the output values of the processes aredetermined and whether or not a predetermined situation has occurred isoutput.

FIG. 35 is a diagram showing one example of a recognition processingsoftware module provided in step S2205. Reference numerals 3501-3505correspond to a module for recognizing the posture of a person, a modulefor detecting an intruder in a predetermined area, a module forrecognizing a person's expressions, a module for recognizingpredetermined movements of a person, and a module for recognizingenvironmental situations (that is, recognition of particular situationssuch as a fire or the like), respectively, which process image dataimaged by the video input unit 410 (and stored in the RAM 405).

The modules operate as middle ware tasks either by time division orserially. In this step, the output values of the modules are output asthe results of analysis of data encoded into a predetermined format. Itshould be noted that these modules may also be implemented as specialhardware modules. In that case, the hardware modules are connected tothe system bus 404 and process the image data stored in the RAM 405 at apredetermined time.

In step S2206, the person who is the object of recognition of thesituation recognized in the process of analyzing the content of thesituation (step S2205) is recognized. Any of the variety of techniquesproposed conventionally can be adapted to that processing relating torecognition of the person which is necessary to this step (e.g., S.Akamatsu: “Research Trends in Face Recognition by Computer”,Transactions of the Institute of Electronics, Information andCommunication Engineers, vol. 80 No. 3, pp. 257-266 (March 1997)). Itshould be noted that the feature parameters needed to identify anindividual are extracted during new registration of the individual asdescribed above (reference numeral 905 shown in FIG. 9).

In step S2207, the reporting control information table is checked and itis determined whether or not a predetermined situation of apredetermined person which should be reported has been recognized, andif so, in step S2208 the process of encoding the content of thesituation is carried out. It should be noted that although in FIG. 25the description of the situation content that is reported is shown aswords expressing a predetermined situation, in actuality a codecorresponding to predetermined code data, not shown, that the process ofanalyzing the content of the situation (step S2206) outputs (that is, acode uniquely specifying a corresponding situation) is recorded in thetable.

Next, a process of encoding the content of the situation (step S2208)converts the situation content into predetermined character informationusing the output from the process of analyzing the content of thesituation (step S2206). This conversion may, for example, provide aconversion table determined in advance, with the character informationobtained from the output of the process of analyzing the content of thesituation (step S2206) and the content of such table.

FIG. 28 is a diagram showing a sample conversion table. For example, asituation recognition processing module R0001 (corresponding to therecognition module 3501 shown in FIG. 35), recognizes and outputs threetypes of situations for a person. A situation recognition processingmodule R0003 (corresponding to the recognition module 3503 shown in FIG.35), recognizes and outputs two types of situations for a person. If apredetermined output is obtained from the recognition processing modules(reference numerals 3501-3505 shown in FIG. 35), the conversion table ischecked and the corresponding predetermined character sequence isoutput. Thus the process of encoding the content of the situation (stepS2208), using the output values (predetermined codes) of the process ofanalyzing the content of the situation (step S2205), obtains characterinformation by checking the conversion table. It should be noted thatthe conversion table is assumed to be recorded in advance in the EEPROM406.

FIG. 36 shows details of the reporting process (step S2209). In thisstep, the person to be notified is determined on the basis of the outputof the process of identifying the place of installation (step S2202),the process of analyzing the content of the situation (step S2205) andthe process of identifying the object of recognition (step S2206), andby checking the reporting control information table (FIG. 25) stored inthe EEPROM 406 in step S3601.

Next, in step S3602, the character information obtained in the situationencoding process (step S2208) is transmitted to the person to benotified. The character information is transmitted via thecommunications interface 408 in accordance with a protocol such aselectronic mail, instant messaging or the like. It should be noted thatthe selection of the reporting destination, in the case of e-mail, isaccomplished by establishing a particular e-mail address for thereporting destination.

It should be noted that, after power is supplied to the main unit, theprocesses of steps S2204-S2209 are executed repeatedly, and when apredetermined situation is recognized, the content of the situation isreported to the person to be notified in that situation.

As can be understood from the foregoing description, according to thepresent embodiment, when a predetermined situation is recognized thecontent of that situation can be easily grasped, and furthermore, theappropriate reporting destination can be notified of the content of thatsituation depending on the place of installation of the device, theobject of recognition and the situation to be recognized.

Fifth Embodiment

FIG. 30 is a diagram showing the structure of a situation monitoringdevice according to a fifth embodiment of the present invention. Thehardware configuration of this embodiment differs from that of the firstembodiment shown in FIG. 4 only insofar as the communications interface408 is different.

Reference numeral 3001 designates a CPU. Reference numeral 3302designates a bridge, which has the capability to bridge a high-speed CPUbus 3003 and a low-speed system bus 3004.

In addition, the bridge 3002 has a built-in memory controller function,and thus the capability to control access to a RAM 3005 connected to thebridge. The RAM 3005 is the memory necessary for the operation of theCPU 3001, and is composed of large-capacity, high-speed memory such asSDRAM/DDR/RDRAM and the like. In addition, the RAM 3005 is also used asan image data buffer and the like.

Furthermore, the bridge 3002 has a built-in DMA function that controlsdata transfer between devices connected to the system bus 3004 and theRAM 3005. An EEPROM 3006 is a memory for storing the instruction dataand a variety setting data necessary for the operation of the CPU 3001.

Reference numeral 3007 designates an RTC IC, which is a special devicefor carrying out time management/calendar management. Reference numeral3009 designate the controls, and is a processor that controls the userinterface between the main unit and the user. The controls 3009 areincorporated in a rear surface or the like of a stand 304 of the mainunit. Reference numeral 3010 designates a video input unit, and includesphotoelectric conversion devices such as CCD/CMOS sensors as well as thedriver circuitry to control such devices, the signal processingcircuitry to control a variety of image corrections, and the electricaland mechanical structures for implementing pan/tilt mechanisms.

Reference numeral 3011 designates a video input interface, whichconverts raster image data output from the video input unit 3010together with a sync signal into digital image data and buffers it. Inaddition, video input interface 3011 has the capability to generatesignals for controlling the video input unit 3010 pan/tilt mechanism.The digital image data buffered by the video input interface 3011 is,for example, forwarded to the predetermined address in the RAM 3005using the DMA built into the bridge 3002.

Such DMA transfer may, for example, be activated using the video signalvertical sync signal as a trigger. The CPU 3001 then commencesprocessing the image data held in the RAM 3005 based on a DMAtransfer-completed interrupt signal that the bridge 3002 generates. Itshould be noted that the situation monitoring device also has a powersupply, not shown.

Reference numeral 3008 a designates a first communications interface,having the capability to connect to a wireless/wire LAN internetprotocol network. Reference numeral 3008 b has the capability to connectdirectly to an existing telephone network or mobile telephone network.In the present embodiment, the reporting medium is selected according tothe object to be recognized and the situation thereof. Specifically,when reporting a normal situation, depending on the degree of urgencythe information is reported using an internet protocol such aselectronic mail, instant messaging or the like. If the situation is anurgent one, then the situation content is reported directly by telephoneor the like.

FIG. 31 is a flow chart illustrating details of the reportingdestination setting process (step S2203) according to the presentembodiment. In this embodiment, compared to the fourth embodimentdescribed above a new reporting medium setting process (step S3105) isadded. The other steps S3101-S3104 are the same as steps S2401-S2404described in the fourth embodiment, and a description thereof isomitted.

FIG. 32 is a diagram showing the content of the reporting controlinformation table used in the present embodiment. In the reportingmedium setting process (step S3105), the reporting medium is setaccording to the place of recognition, the object of recognition and thecontent of the situation. In the case of FIG. 32, it is specified thatreporting is to be “by telephone” for such extremely urgent situationsas “Has person fallen?” and “Suspicious person detected”. By contrast,“by instant messaging” is specified for such situations of intermediateurgency as “Is person in pain?”, “Has person put something in hismouth?” and “Is person in a prohibited area?”, and “by e-mail” isspecified for such situations of lesser urgency as “Entry/exitconfirmed”.

The information set in step S3105, as with the fourth embodimentdescribed above, is then recorded in the EEPROM 3005 as the reportingcontrol information table.

In the situation content encoding process (step S2208) of the presentembodiment, the situation content is encoded according to the reportingmedium set in the reporting medium setting process (step S2203). Forexample, character information is encoded if “instant messaging” or“e-mail” are set as the reporting medium, and voice information isencoded if “telephone” is set as the reporting medium. The encoding ofvoice information generates voice data corresponding to the charactersequence shown in the table shown in FIG. 28 by a voice synthesisprocess, not shown. It should be noted that such voice data may becompressed using high-efficiency compression protocols such as ITUstandard G.723 or G.729. The voice information thus generated is thentemporarily stored in the RAM 3005 or the like.

FIG. 37 is a diagram illustrating details of the reporting process(S2209). In step S3701, the reporting control information table (FIG.32) stored in the EEPROM 3006 is checked and a predetermined reportingdestination is determined according to the output of the process ofidentifying the place of installation (step S2202), the output of theprocess of identifying the object of recognition (step S2206) and theoutput of the process of analyzing the content of the situation (stepS2205).

Next, in step S3702, similarly, the reporting control information tableis checked and the reporting medium determined. Encoded informationexpressing the content of the situation is then transmitted to thereporting destination selected in step S3702 through the selectedreporting medium (3008 a or 3008 b). In other words, if “instantmessaging”, “e-mail” or the like is selected as the reporting medium,the report content is transmitted according to internet protocol throughthe first communications interface 3008 a. If “telephone” is selected asthe reporting medium, then the telephone of the predetermined reportingdestination is automatically called and after ringing is confirmed thevoice data held in the RAM 3005 is transmitted as direct audio signalsthrough the second communications interface 3008 b.

Thus, according to the present embodiment, it is possible to notify apredetermined reporting destination by reporting medium selectedaccording to the situation, achieving a reporting capability suited tothe degree of urgency.

Sixth Embodiment

FIG. 33 is a diagram showing the outlines of a processing flow performedby a situation monitoring device according to a sixth embodiment of thepresent invention. The flow chart is a program loaded in the RAM 3005and processed by the CPU 3001. The hardware configuration of thesituation monitoring device according to the present embodiment is thesame as that of the fifth embodiment, and therefore a description isgiven only of the difference between the two.

FIG. 33 is a flow chart illustrating details of the reportingdestination setting process (step S2203) of the present embodiment. Inthis embodiment, in contrast to the reporting destination settingprocess of the fifth embodiment, a reporting determination time settingprocess step (S3306) is newly added. The remaining steps S3301-S3305 areeach the same as steps 3101-S3105 described in the fourth embodiment,and thus a description of only the difference therebetween is given.

FIG. 34 is a diagram showing one example of a reporting controlinformation table according to the present embodiment. In the event thattime information corresponding to recognition situations is set and apredetermined situation is recognized, that recognized time isdetermined and the content of the recognition situation is reported tothe reporting destination in accordance with the time. For example, inthe case of location code P0003, if an intruder is detected between thehours of 0800 and 2400, the system is set to notify the mother byelectronic mail. By contrast, if an intruder is detected between thehours of 2400 and 800 under the same conditions, the system is set tonotify the security company. The information set in step S3306, as withthe fourth embodiment, is recorded in the EEPROM 3006 as a reportingcontrol information table.

FIG. 38 is a flow chart illustrating details of the reporting process(step S2209) according to the present embodiment. In step S3801, thetime that a predetermined situation is recognized is obtained from theRTC 3007. In step S3802, based on the place of recognition, the personwho is the object of recognition, the recognition situation and the timeobtained in step S3801, the reporting control information table (FIG.34) stored in the EEPROM 3006 is checked and a predetermined reportingdestination determined.

Furthermore, in step S3803, the reporting control information table issimilarly checked and a predetermined reporting medium determined. Instep S3804, data encoded in step S2208 showing the content of thesituation is transmitted to the reporting destination determined in stepS3803 through reporting medium determined in step S3804.

As can be understood from the foregoing description, with the presentembodiment, based on the time when a predetermined situation isrecognized, it is possible to report to more appropriate reportingdestinations using more appropriate reporting medium.

It should be noted that although the foregoing embodiments are describedin terms of a person as the object of recognition, the present inventionis not limited thereto and the object of recognition may be an animal, aparticular object or anything else. For example, in the case of aparticular object, situations such as that object “Has been moved from apredetermined position” or “Has gone missing” may be recognized andreported. The recognition of movement or presence/absence can be easilyaccomplished by the use of pattern matching techniques proposedconventionally.

Although in the foregoing embodiments the reporting control informationtable specifies the reporting destination and reporting medium dependingon the place of installation of the device and the object ofrecognition, the time and the situation, the present invention is notlimited thereto. Depending on the purpose, a table that designates thereporting destination or the reporting medium according to at least oneof the place of installation, the object of recognition and the time aswell as the situation may be provided.

Although the foregoing embodiments are described in terms of the processof analyzing the content of the situation by providing a plurality ofsituation recognition processes and utilizing the output of thoseprocesses to analyze the situation content, the present invention is notlimited thereto and any method may be used. For example, a moregeneralized recognition algorithm may be installed and all targetsituations recognized.

Although the foregoing embodiments are described in terms of encodingthe results of the process of analyzing the content of the situation aspredetermined character sequences or audio information, the presentinvention is not limited thereto and these results may be converted intoother types of information. For example, such information may beconverted into diagrammatic data that expresses the informationschematically, and such diagrammatic data transmitted as reporting data.In addition, instead of reporting over a network, a method may be usedin which light patterns from a predetermined light source are reportedas warning information.

Although the fourth embodiment described above is described in terms ofusing video information to recognize the place of installation of thedevice and the situation of the object of recognition, the presentinvention is not limited thereto and sensing information other thanvideo information may be used to recognize the situation. Furthermore,situations may be recognized using a combination of video informationand other sensing information. As other sensing information it ispossible to use a variety of sensing technologies such as audioinformation, infrared ray information and electromagnetic information.

Although the foregoing embodiments are described in terms of the mediumthat report a change in the situation of the object of recognition asinternet mail, instant messaging and telephone, etc., the presentinvention is not limited thereto and other medium may be used asnecessary.

Although the foregoing embodiments are described in terms ofestablishing the reporting control information table using the controls409, alternatively a network may be used to set the parameters necessaryfor operation. In this case, the main unit may have a HTTP (Hyper TextTransfer Protocol) server capability, for example, and provide aWeb-based user interface to the user through the communicationsinterface 3008. The HTTP server is incorporated as one type of middleware, and activates a predetermined parameter setting program inresponse to operation from a remote location based on HTTP.

In this case, the user can set the parameters necessary for operation ofthe main unit from an ordinary terminal such as a mobile telephone, aPDA or a personal computer, and furthermore, such setting operations canbe carried out from a remote location.

Although the foregoing embodiments are described in terms of executingall processing such as the recognition processing using a processorbuilt into the main unit, the present invention may be implemented, forexample, in combination with an external processing device such as apersonal computer or the like. In this case, only the reading in ofimage data is accomplished using a specialized device, with theremaining processes, such as image recognition and communications,implemented using personal computer resources.

By using a wireless interface such as BlueTooth, for example, or a powerline communications interface such as HPA (Home Power Plug Alliance) orthe like to connect the specialized device and the personal computer,the same convenience can be achieved. This sort of functionallydispersed situation monitoring system can of course be achieved not onlywith the use of a personal computer but also with the aid of a varietyof other internet appliances as well.

Although the foregoing embodiments are described in terms ofimplementing the present invention by software processing using a CPU,the present invention is not limited thereto and may, for example, beimplemented by special hardware processing as well. In that case, thealgorithm for situation recognition corresponds to object data thatdetermines the internal circuitry of an FPGA (Filed Programmable GateArray) or object data that determines the internal circuitry of areconfigurable processor. The system control processor loads the datafrom the EEPROM 406 or a server device connected to the network and thelike into the special hardware. The special hardware then commencesrecognition processing of a predetermined algorithm according to theobject data that has been loaded.

Although the foregoing embodiments are described in terms of using acamera having a mechanical control structure (a so-called pan/tiltcamera), the present invention is not limited thereto and may, forexample employ a wide-angle camera instead. In that case, the object ofrecognition is not supplemented mechanically but instead an equivalentprocess can be implemented using image data acquired at wide angles.

Other Embodiments

It should be noted that the present invention can be adapted to a systemcomprised of a plurality of devices (for example, a host computer, aninterface device, a reader, a printer and so forth) or to an apparatuscomprised of a single device.

In addition, the invention can be implemented by supplying a softwareprogram, which implements the functions of the foregoing embodiments,directly or indirectly, to a system or apparatus, reading the suppliedprogram code with a computer (or CPU or MPU) of the system or apparatus,and then executing the program code.

In this case, the functions of the foregoing embodiments are implementedby the program code itself read from the storage medium, and the storagemedium storing the program code constitutes the invention.

Examples of storage media that can be used for supplying the programcode are a floppy disk (registered trademark), a hard disk, an opticaldisk, a magneto-optical disk, a CD-ROM, a CD-R, magnetic tape, anonvolatile type memory card, a ROM or the like.

Besides those cases in which the aforementioned functions according tothe embodiments are implemented by executing the program code read bycomputer, the present invention also includes a case in which an OS(operating system) or the like running on the computer performs all orpart of the actual processing according to the program codeinstructions, so that the functions of the foregoing embodiments areimplemented by this processing.

Furthermore, after the program read from the storage medium is writtento a function expansion board inserted into the computer or to a memoryprovided in a function expansion unit connected to the computer, a CPUor the like mounted on the function expansion board or functionexpansion unit performs all or part of the actual processing so that thefunctions of the foregoing embodiment can be implemented by thisprocessing.

The present invention is not limited to the above embodiments andvarious changes and modifications can be made within the spirit andscope if the present invention. Therefore, to apprise the public of thescope of the present invention, the following claims are made.

CLAIM OF PRIORITY

This application claims priority from Japanese Patent Application No.2004-167544 filed on Jun. 4, 2004 and Japanese Patent Application No.2005-164875 filed on Jun. 3, 2005, the entire contents of which arehereby incorporated by reference herein.

1. A situation monitoring device comprising: place recognition means forrecognizing a place of installation where the device is installed;information holding means for holding relational information relatingthe place of installation and a situation to be recognized;determination means for determining a predetermined situation to berecognized, in accordance with recognition results by the placerecognition means and the relational information; situation recognitionmeans for recognizing the predetermined situation determined by thedetermination means; and communications means for reporting therecognition result of the predetermined situation recognized by thesituation recognition means to the user.
 2. A situation monitoringdevice comprising: place recognition means for recognizing a place ofinstallation where the device is installed; target object recognitionmeans for recognizing the target object; information holding means forholding relational information relating the place of installation withthe target object to be recognized and a situation to be recognized;determination means for determining the target object to be recognizedand a predetermined situation to be recognized, in accordance withrecognition results by the place recognition means and the relationalinformation; situation recognition means for recognizing that the targetobject determined by the determination means is in the predeterminedsituation determined by the determination means; and communicationsmeans for reporting the recognition result of the predeterminedsituation recognized by the situation recognition means to the user. 3.A situation monitoring device comprising: target object recognitionmeans for recognizing a particular target object; information holdingmeans for holding relational information relating the target object anda situation to be recognized; determination means for determining apredetermined situation to be recognized, in accordance with recognitionresults by the target object recognition means and the relationalinformation; situation recognition means for recognizing that the targetobject recognized by the target object recognition means is in thepredetermined situation determined by the determination means; andcommunications means for reporting the recognition result of thepredetermined situation recognized by the situation recognition means tothe user.
 4. The situation monitoring device according to claim 1,wherein the situation recognition means comprises an acquisition meansfor acquiring image data, and recognizes the predetermined situationfrom the acquired image data.
 5. The situation monitoring deviceaccording to claim 1, wherein the place of installation recognitionmeans comprises an acquisition means for acquiring image data, andrecognizes the place of installation from the acquired image data. 6.The situation monitoring device according to claim 1, wherein the placerecognition means commences a process of recognition of the place ofinstallation when a predetermined condition is detected.
 7. Thesituation monitoring device according to claim 6, wherein thepredetermined condition is a change in the sensed image.
 8. Thesituation monitoring device according to claim 6, wherein the placerecognition means comprises a sensor for detecting movement of thedevice and the predetermined condition is a change in such sensorinformation.
 9. The situation monitoring device according to claim 6,further comprising controls for inputting parameters necessary foroperation of the device, and the predetermined condition is a particularinput by the user to the controls.
 10. The situation monitoring deviceaccording to claim 6, wherein the predetermined condition is power on ofthe device.
 11. The situation monitoring device according to claim 6,wherein the predetermined condition is a time determined in advance. 12.The situation monitoring device according to claim 1, wherein thecommunications means further reports to the user that a shift in theplace of installation has been recognized by the place recognitionmeans.
 13. The situation monitoring device according to claim 1, whereinthe communications means further reports to the user that the object tobe recognized has changed.
 14. The situation monitoring device accordingto claim 1, further comprising controls for inputting parametersnecessary for operation of the device and an interface prompting theuser to update the relational information under predetermined conditionsis displayed on the controls.
 15. The situation monitoring deviceaccording to claim 14, wherein the predetermined condition is the placerecognition means recognizing a shift in the place of installation. 16.The situation monitoring device according to claim 14, wherein thepredetermined condition is the place recognition means recognizing aplace of installation that is not registered in the relationalinformation.
 17. The situation monitoring device according to claim 14,wherein the predetermined condition is the target object recognitionmeans recognizing a target object that is not registered in therelational information.
 18. The situation monitoring device according toclaim 1, wherein a situation of a default determined in advance isdetermined by the determination means when the place recognition meansrecognizes a place of installation that is not registered in therelational information.
 19. The situation monitoring device according toclaim 3, wherein a situation of a default determined in advance isdetermined by the determination means when the target object recognitionmeans recognizes a target object that is not registered in therelational information.
 20. The situation monitoring device according toclaim 1, wherein the situation recognition means recognizes a situationin accordance with an order of priority determined in advance when aplurality of target objects exist for a recognized location.
 21. Thesituation monitoring device according to claim 1, wherein the device hasa configuration dispersed in a main part and a peripheral part, andinformation for recognizing the place of installation with the placerecognition means is held in the peripheral part.
 22. The situationmonitoring device according to claim 1, wherein the place recognitionmeans further comprises external communications means for communicatingwith an external device disposed adjacent to an external apparatus orthe main unit, and recognizes the place of installation according toinformation emitted by the external apparatus or information held by theexternal device.
 23. The situation monitoring device according to claim1, further comprising controls separate from the main unit and controlscommunications means for communicating with the controls, wherein thesetting of parameters necessary for operation of the device is carriedout using the controls.
 24. A situation monitoring device according toclaim 1, further comprising connection means for connecting to a networkand a server device, wherein the setting of parameters necessary foroperation of the device is carried out from the external apparatus usingthe server device.
 25. The situation monitoring device according toclaim 24, wherein the server device is a HTTP (Hyper Text TransferProtocol) server.
 26. A situation monitoring system comprising: thesituation monitoring device according to claim 1; and connection meansfor connecting to a network, wherein a processing algorithm executed bythe situation recognition means is held in another external apparatusconnected to the network.
 27. A situation monitoring device comprising:situation analyzing means for analyzing a situation of a target object;discrimination means for identifying a predetermined situation fromoutput from the situation analysis means; situation encoding meansconfigured to convert the situation into a predetermined signal based onthe output from the situation analysis means; and communications meansfor reporting the output of the situation analysis means to the userusing the situation encoding means.
 28. The situation monitoring deviceaccording to claim 27, wherein the predetermined signal is at least oneof diagrammatic symbol information, character information, voiceinformation and light-emitting pattern information expressing thesituation content, encoded by a predetermined method.
 29. The situationmonitoring device according to claim 28, further comprising controlmeans for selecting at least one reporting destination from amongpredetermined reporting destinations based on the output of thesituation analysis means.
 30. The situation monitoring device accordingto claim 29, further comprising target object identification means foridentifying a target object, wherein the control means selects at leastone reporting destination from among predetermined reportingdestinations based on output of the target object identification meansand the output of the situation analysis means for that target object.31. The situation monitoring device according to claim 29, furthercomprising location identification means for identifying a location,wherein the control means selects at least one reporting destinationfrom among predetermined reporting destinations based on output of thelocation identification means and the output of the situation analysismeans.
 32. The situation monitoring device according to claim 29,further comprising: target object identification means for identifying atarget object; and location identification means for identifying alocation, wherein the control means selects at least one reportingdestination from among predetermined reporting destinations based on theoutput of the target object identification means and the output of thelocation identification means as well as the output of the situationanalysis means.
 33. The situation monitoring device according to claim29, further comprising time management means for managing time, whereinthe control means selects at least one reporting destination from amongpredetermined reporting destinations based on output of the timemanagement means and the output of the situation analysis means.
 34. Thesituation monitoring device according to claim 27, comprising aplurality of communications means and control means for selecting atleast one communications means from among the plurality ofcommunications means based on the output of the situation analysismeans.
 35. The situation monitoring device according to claim 34,further comprising target object identification means for identifying atarget object, wherein the control means selects at least onecommunications means from among the plurality of communications meansbased on output of the target object identification means and the outputof the situation analysis means for the target object.
 36. The situationmonitoring device according to claim 34, further comprising locationidentification means for identifying a location, wherein the controlmeans selects at least one communications means from among the pluralityof communications means based on output of the location identificationmeans and the output of the situation analysis means.
 37. The situationmonitoring device according to claim 34, further comprising: targetobject identification means for identifying a target object; andlocation identification means for identifying a location, wherein thecontrol means selects at least one communications means from among theplurality of communications means based on the output of the targetobject identification means, the output of the location identificationmeans and the output of the situation analysis means.
 38. The situationmonitoring device according to claim 34, further comprising a timemanagement means for managing time, wherein the control means selects atleast one reporting destination from among predetermined reportingdestinations based on output of the time management means and the outputof the situation analysis means.
 39. The situation monitoring deviceaccording to claim 27, wherein the situation analysis means comprisesimage data acquisition means for acquiring image data, and analyzes thesituation from the image data acquired by the image data acquisitionmeans.
 40. The situation monitoring device according to claim 30,wherein the target object identification means for identifying a targetobject comprises image data acquisition means for acquiring image data,and identifies the target object using the image data acquired by theimage data acquisition means.
 41. The situation monitoring deviceaccording to claim 31, wherein the location identification means foridentifying a location comprises image data acquisition means foracquiring image data, and identifies the location using the image dataacquired by the image data acquisition means.
 42. The situationmonitoring device according to claim 27, further comprising: connectingmeans for connecting to a network; and a server function, whereinparameters necessary for operation of the device can be set from anexternal terminal device through the server function.
 43. The situationmonitoring device according to claim 42, wherein the parameters arecontrol parameters for the control means that selects the reportingdestination.
 44. The situation monitoring device according to claim 42,wherein the parameters are control parameters for the control means thatselects the reporting means.
 45. The situation monitoring deviceaccording to claim 42, wherein a HTTP (Hyper Text Transfer Protocol)server implements the server function.